Sparse representation of solutions of Kronecker 
product systems * 

Sadegh Jokar Volker Mehrmann^" 

31.07.08 

Abstract 

Three properties of matrices: the spark, the mutual incoherence 
and the restricted isometry property have recently been introduced 
in the context of compressed sensing. We study these properties for 
matrices that are Kronecker products and show how these properties 
relate to those of the factors. For the mutual incoherence we also 
discuss results for sums of Kronecker products. 
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1 Introduction 

In this paper we discuss the computation of sparse solutions of underdeter- 
mined linear systems 

Ax = b, 

where A G M™'", with m < n is given as a Kronecker product, i.e. 

A = Ai0A2(^...(^An, GM""-"Si = l,...,iV, (1) 
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or as a sum of Kronecker products 

M 

Since the solution is typically non-unique it is an important topic in many 
applications, in particular in optimal signal recovery and in compressed sens- 
ing, see e.g. [H El H EJ El El [111 [El EQI to find the sparsest solution, 

min ll^llo, s.t. Ax = b, (3) 

where ||x||o denotes the number of nonzero entries of a vector x, see Section^ 
In general, the problem of finding the sparsest solution is known to be 
NP-hard [22]. However, in the context of compressed sensing, conditions 
have been derived on the size of the support of x, i.e. the number of nonzero 
elements of x, that allow one to compute the sparsest solution using £i-mini- 
mization via the so called basis pursuit algorithm [3l El El El UHl [HI [12]) i-e, 
by computing 

min ||x||i, s.t. Ax = b, (4) 

where ||rE||i = X^Ja^il- 

Sufficient conditions for this approach to work are that some properties 
of the matrix A called spark [10\ I25j . mutual incoherence [3 |T2] or the 
restricted isometry property (RIP) [2l[3ll31 are restricted. We will introduce 
these properties in Section [2l 

For general matrices it is possible (though expensive) to determine the 
mutual incoherence, while analyzing the spark or the restricted isometry 
property is difficult. If, however, the matrix A has the form ^ then we 
show in Section [3] that these properties can be easily derived from the corre- 
sponding properties of the factors. For the mutual incoherence we can also 
extend these results to matrices of the form ([2]). 

2 Notation and preliminaries 

For m, n G N, where N = {1, 2, . . . }, we denote by M™"'" the set of real mxn 
matrices, by In the n x n identity matrix, and by (•, •) the Euclidean inner 
product in M". For 1 < p < oo, the ^p-norm of x G M" is defined by 

n 

. ^ 1 

W^Wp -~ ( ' 
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with the special case 



ie{i,...,n} 



if p = oo. Finally, for x G R", we introduce the notation 

||x||o := #supp(x), 

where supp(j;) := {j £ {1, . . . ,n} : xj ^ 0} is the support of x. Note that 
||-||o is not a norm, since for a 7^ we have ||ax||o = ||a;||o- We use the term 
k-sparse for all vectors x such that ||x||o < k. 

Definition 2.1 \M The Kronecker product of A = [oij] G W'^ and 
B = [bij] £ M^'* is denoted hy A® B and is defined to be the block matrix 



A^B :-- 



aiiB 



a„iB 



ai,qB 



a"P,qB 



G 



ppr,qs 



Let C = [ci - ■ ■ Cr] £ M'^''' with columns Cj G M'', 1 < i < r. Then, 

Cl 



vec(C7) := 



It is well known [21j that the matrix equation AXB = C, with matrices of 
appropriate dimensions, is equivalent to the linear system 

(5^ ^)vec(X) = vec(C). 

Furthermore, using the perfect shuffle permutation matrices IIi, 112, we have 
that ni(^ (g) B)U2 = B (g> A, see 

As our first special property we introduce the spark of a matrix. 



2 < m < n have 
1, . . . , 71. The spark 



Definition 2.2 [MIS^ Let A = [ai,...,a„] G 
columns ai that are normalized so that \\ai\\2 = I, i 
of A, denoted as spark(^) is defined as the cardinality of the smallest subset 
of linearly dependent columns of A. 

In other words, if all r-dimensional subsets of column vectors of A are 
linearly independent, but there exists a subset of r + 1 columns that are 
linearly dependent, then spark(j4) = r + 1. For convenience, if m = n = 1, 
we define spark(j4) := 1, and in the case where m = n > 2 and A is 
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invertible, we set spark(^) := n + 1. In general the spark and the rank of a 
matrix A £ W^'"" with m > 2, are related via 

2 < spark(A) < rank(^) + 1. 



Example 2.3 If 

[10 1 1 
^" [ 1 1 -1 

then spark(74) = rank(j4) + 1 = 3. On the other hand, if 



A 



10 1-1 

011 



then spark(A) = 2. 



The quantity spark(^) can be used to derive necessary and sufficient condi- 
tions for the existence of sparse solutions. 

Lemma 2.4 110[ \16^ Consider the linear system Ax = b with A £ M*"'", 
m < n. A necessary and sufficient condition for the linear system Ax = b 
to have a unique k-sparse solution x is that k < spark(j4)/2. 

The second property that we study is the mutual incoherence. 

Definition 2.5 fW^ Let A = [ai,...,a„] G M'"'", m < n have columns 
ai that are norm,alized so that ||ai||2 = Ij i = l,...,n. Then the mutual 
incoherence Ai{A) is defined by 

T 

M{A) := max |(aj, aj)| =max|(^ 

Note that, since the columns of A are normalized, by the triangle inequality 
we always have Ai{A) < 1. On the other hand, if A has orthonormal 
columns, then A4{A) = 0. 

We have the following lower bound for M{A). 

Lemma 2.6 1241 Suppose that A £ M"^'", m < n has columns ai that are 
normalized so that \\ai\\2 = 1, i = 1, . . . ,n and suppose further that A has 
full row rank. Then 

. , / ,N I n — m 
M{A) > 



m{n — 1) 
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The following lemma relates the sparsest solution as defined in ([3]) and the 
^i-solution as defined in (jH) of the linear equation Ax = 6 in terms of the 
mutual incoherence of a matrix A. 

Lemma 2.7 !11U[ IJ5J / Suppose that A G M"*'", m < n has columns that 
are normalized so that \\ai\\2 = 1, i = l,...,n. Ifb is a vector such that the 
equation Ax = b has a solution satisfying 

II II ^ ^ + M{A) 

then the li-norm minimal solution in ^ coincides with the iQ-minimal so- 
lution in ([^j. 

Remark 2.8 Consider matrices of the form A = where $ and ^ 

have orthonormal columns. If the sparsest solution x of Ax = b satisfies 

Ikllo < 



M{A) ' 

then it has been shown in that the solutions of the £i-norm minimization 
problem and io-norm minimization problem coincide. 

The third quantity that is important in the context of sparse recovery 
and compressed sensing is the restricted isometry property. 

Definition 2.9 13, \^ Let A = [m, . . . , a„] G M™'", m < n have 
columns ai that are normalized so that ||ai||2 = 1, i = 1, . . . , n. 

The A:-restricted isometry constant of A is the smallest number 6k such 

that 

{l-5k)\\x\\l<\\Ax\\l<{l + 6k)\\x\\l 
for all X £ M" with \\x\\q < k. 

The ^-restricted isometry property requires that every set of columns of 
cardinality less than or equal to k approximately (with an error 5k) behaves 
like an orthonormal basis. 

The following lemma gives the relation between the sparsest solution (as 
defined in ([3])) of a linear system Ax = b and the ^i-solution as defined in 
@ in terms of the ^-restricted isometry constant. 
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Lemma 2.10 /^/ Let A = [ai, . . . , a„] G M™''", m < n have columns ai that 
are normalized so that ||ai||2 = 1, "i = 1, . . . , n. 
Suppose that 

52k < 72-1. 

T/ien for all k-sparse solution vectors x of Ax = b the solution of ^ is 
equal to the solution of 

After introducing the concepts of spark, mutual incoherence and k- 
restricted isometry property, in the next section we analyze these concepts 
for Kronecker product matrices. 

3 Sparse representation and Kronecker Products 
of Matrices 

In this section we study sparse solutions for linear system Ax = b, where 
the matrix A is given as a Kronecker product ([1]). 

Our first result characterizes spark(^ B) in terms of spark(A) and 
spark(i?). Note that if A,B have normalized columns then A® B has 
normalized columns as well. 

Theorem 3.1 Let A = [ai, . . . , a,] G W''i and B = . . . , 6^] G W^' 
be rank- deficient matrices with normalized columns, i.e., ||aj||2 = 1, i = 
1, . . . ||6i||2 = 1, i = 1, . . . , s. Then 

spark(^ ® B) = spark(i? ® A) = min{spark(^), spark(i?)}. (5) 
If A is an invertible matrix and B is rank- deficient matrix, then 

spark(^ B) = spark(5). (6) 
If both A and B are square and invertible then 

spark(^ (g)B) = (spark(A) - l)(spark(5) -l) + l = qs + l. 

Proof. Using the fact that {B (g) A)vec{X) = ni(y4 (g) B)U2vec{X) and 
||vec(X)||o = ||n2vec(X)||o, we have spark(j4 B) = spark(i? A). 

Consider first the case that A and B are rank-deficient. By the definition 
of spark(i?), there exists a vector y G M** with ||y||o = spark(i?) such that 
By = 0. With 

X=[y ••• ], 
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we have that {A B)vec{X) = and ||vec(X)||o = ||y||o = spark(i?). This 
means that spark(^(K)i?) < spark(i3). Using that spark(A(K>i?) = spark(i?C3 
^4) and that also A is rank-deficient, we can apply the same argument as 
before and get spark(j4 B) < spark(j4). Therefore, 

spark(^ fSi B) < min{spark(A), spark(i?)}. (7) 

Let C = A<Si B, then every column of C has the form Cj = by. . To 
prove equality in ([5]), we assume w.l.o.g. that 

spark(5) < spark(^). (8) 

Then by ([7|) we have spark(j4 B) < spark(i3). Suppose now that 

spark(^ ^ B) = £ < spark(S). (9) 

This implies, in particular, that any set of i columns of B is linearly inde- 
pendent, while there exist scalars Ai, . . . , not all and indices ui, . . . ,Ui 
where Ui / Uj for all i ^ j, and vi, . . . ,Vi such that 

e I 
^(a„^ ® bv^)\j = ^(Aja„^.) ® b^^ = 0. 
i=i i=i 

In this sum there may occur repeated copies of vectors 6j, so without loss 
of generality we may assume the indices Vi are numbered so that 

Vi = ■■■ =Vki < ■Ufci + l = ■ ■ ■ = < ■ ■ ■ < Vkt-i + l = ■■■ =Vkf 

^ V ' ^ V ' ^ ^ ' 

91 92 gt 

Therefore, we have 

ki k2 kt 

{J2 )^^9i+i Yl ^^"'^^ ) ® + • • • + ( Yl = 0' (10) 

j=l j=ki+l j=kt-i+l 

where kt = L Since bg^^, . . . , bg^ are linearly independent, it follows that for 
all 1 < i < t we have 

j=fci_i+i 

where fco = 0. This contradicts the assumption in Q that 

t < spark(i?) < spark(A), 
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because the uj are pairwise distinct and at least one of the coefficients Xj is 
nonzero. 

Now suppose that A is invertible and B is rank-deficient. Then with 
the same argument as above, we have spark(^ < spark(i3). Let X = 
[xi, . . . ,Xq] / 0, such that ||vec(X)||o = spaxk{A^ B) and {A(^ B)Yec{X) = 

T 

0. This imphes that BXA = 0, and, since A is invertible we have BX = 0, 
while on the other hand X 0. Thus there exists at least one index i such 
that Xi ^ and Bxi = 0. Hence, 

spark(i?) < ||xj||o < ||vec(X)||o = spark(^ (g) S), 

and therefore spark(yl <^ B) = spark(i?). 

For the case where both A and B are invertible, Af^iBis invertible as 
well, see (2T] . Therefore, 

spark(A ^ B) = rank(^ ^ B) + 1 = qs + I. 

□ 

We immediately have the following corollary of Theorem 13. 1[ 

Corollary 3.2 Consider rank- deficient matrices {Ai}^-^ with normalized 
columns. Then 

sparkfAi ® . . . (g) Am) = min |spark(A)|. 

l<i<N 

By combining Lemma 12.41 and Corollary 13.21 we get the following Corollary: 



Corollary 3.3 Consider a linear system {Ai ... <Si Aj\f)x = b with rank- 
deficient matrices Ai G MPi'S* that have normalized columns. A necessary 
and sufficient condition for this linear system to have a unique k-sparse 
solution X is that 

min {spark(^j)} 



Remark 3.4 Corollary \3.3\ is saying that if one of the matrices Aj has 
small spark then we can only uniquely recover vectors of the sparsity up to 
spark(Aj)/2 in the linear system {Ai ® . . . Afyf)x = b. 

Similar to the analysis of spark(A <Si B), we can also obtain an estimate 
M{A(S) B) in terms of M{A) and M{B). 
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Theorem 3.5 Consider matrices A = [ai,...,ani] € M™-!'"! and B = 
[61, ... , ftnj] G ]R™2,n2 ^nJi normalized columns. Then 

M{A0B) =ma^{M{A),M{B)}. 

Proof. Suppose that C = A ^ B and C = [ci •••c„], where Cj G W^, 
m = mim2 and n = nin2- Then we have A4(C) = maxj^j|(cj, Cj)|. Since 
Cj = Op (X" bq and Cj = a,, (g) 6^ for some p,q,r,s, using properties of the 
Kronecker product [21j, we have 

{ci,Cj) = {ap bg,ar bg) = {ap,ar) ■ {bq,bs). (11) 

By Definition 12.51 and (llip we then have 

M{C)=MiA0B) = max \{ap, Ur) ■ {bg,b,)\ 

(p.g)7^(f'.s) ^2) 

On the other hand, since the matrices A and B have normahzed columns, 
we have 

\{ap,ar) ■ {bq,bs)\ < \{ap,ar)\, 

and similarly 

\{ap,ar) ■ {bq,bs)\ < \{bq,bs)\. 
Therefore, from (1121) we have 



M{A0B) = max {|(ap,o,)|,|(6,,6,)|} 

= max{max| (op, a^)], max| {bq, bs)\} 
= max{M{A),MiB)}. 

□ 

A direct consequence of this theorem is the following Corollary. 

Corollary 3.6 Consider matrices {Ai}^^ with normalized columns and let 
A = Ai(^ . . . (gi An- Then, 

M{A) = max M{Ai). 

l<i<n 

Corollary 13.61 shows that if one of the matrices Ai has a large mutual inco- 
herence, then it will dominate the mutual incoherence of A, regardless of all 
the other factors in the Kronecker product. 

We also have a result that relates the A:-restricted isometry constant of 
6^^^ to those of 5^ and . 
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Theorem 3.7 Let A G M^''' and B £ W''^ have normalized columns. Then 

'5.^^^ = 5r^>maxR^<5f}. (13) 

Proof. Using the fact that B ^ A = Ili{A ® B)Il2, where Hi and 112 are 
permutation matrices we have 

||vec(X)||i = ||n2vec(X)||i, 

and 

\\{B (g) A)veciX)\\l = \\Ui{A<S)B)U2vec{X)g = \\{A ® B)iU2vec{X))\\l 

Therefore 5^®'^ = ^k^^- To prove the assertion, it is sufficient to prove 
that 5^®^ > 5f , the proof that 6^'^^ > 6^ fohows analogously. We know 
that 5^ is the smallest constant such that, for all x with ||3;||o < k, we have 

{l-S^)\\xg<\\Bxg<{l + 6^)\\x\\l 

For any x with ||x||o < k, we can construct the matrix X = [ x • • • ], 
with ||vec(X)||o < k. Since A has normalized columns, we have 



||(^ i?)(vec(X))||2 = ay\Bx\\l = \\Bx\\l (14) 

1=1 

and 

||vec(X)||2 = (15) 
On the other hand 5^'^^ is the smallest constant such that 

(1 - 5,^«^)||vec(X)||2 <\\{A® B){vec{X))\\l < (1 + 5,^«^)||vec(X)||2, 

and for the special class of /c-sparse vectors vec(X) from (jl4p and (jl5p we 
have 

(1 - 5^)hec{X)\\l <\\{A<^ B){vec{X))\\l < (1 + 5^)hec{X)\\l 

where 5^ is the smallest constant for this special class of /c-sparse vectors. 
Therefore, for general /c-sparse vectors, we have 

5t^^ > 5f . 
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Remark 3.8 Note that for k = 2, equality holds in U3\) . since for a given 
normalized matrix A, we have 82 = M{A). Therefore, by Theorem 13.51 it 
follows that 

^2^^^ =max{52^,(^f}. 

For k > 3, however the inequality may be strict. For example if A = [I2 H2] 
and B = [I4 Hi\, where In is the identity matrix of order n and Hn is 
the Hadamard matrix of order n, see e.g. [I3 [1^, then 6^^^ = 1.0545 > 
max{l, -^}- Here the k-restricted isometry constants of these matrices were 
calculated using the singular value decomposition for all submatrices consist- 
ing of 3 columns. 

We have the obvious corohary. 

Corollary 3.9 Suppose that matrices Ai for i = 1, . . . , N have normalized 
columns. Then 

> max {6^^}. 

l<i<N 

According to Lemma [2.10l if the restricted isometry constant 62k is small 
enough {52k < — l)i then one can recover all A:-sparse solutions using 
£i-minimization. On the other hand, Corollary 13.91 implies that if the k- 
restricted isometry constant 5k of A is small (for example less than 1/2), 
then A can not be written as a Kronecker product of matrices Ai with 
smaller sizes. 



4 Sums of Kronecker products 

In many applications, in particular in finite difference or finite element dis- 
cretizations of partial differential equations in more than one space dimen- 
sion [23] , linear systems with matrices that are sums of Kronecker products 
arise. 

It is then an obvious question whether the spark, the mutual incoherence 
and the A:-restricted isometry property for sums of Kronecker products can 
be related to that of the summands. 

Unfortunately, in general we do not have a nice relation between spark(A+ 
B) and spark(^), spark(B). 

Example 4.1 Let En denote the n x n matrix of all ones. If 

A=[ I5 E5](g) 



1 1 
1 -1 
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and 



B 



[1 2 3 4 5] [1 2 3 4 5] h 







" 1 1 " 


h 




-1 



then 5 = spark(^ + B) > spark(^) + spark(5) =2 + 2. 

On the other hand if A = I2 h and A + B = \{E2 ® E2) then 2 = 
spark(A + B) < spark(A) + spark(B) = 5 + 5. 

For the mutual incoherence the situation is better. We introduce the 
following concept of diagonal and off-diagonal mutual incoherence. 

Definition 4.2 Suppose that A = [ai,...,a„], B = [61,..., 6„] G R"^'", 
m < n, have normalized columns. Then the off-diagonal mutual incoherence 
Mooi^, B) of A and B is defined via 

MoDiA,B) := max|(ai,6j)| 

and the diagonal mutual incoherence i?) of A and B is defined via 

Md{A,B) := max|(ai,6i)|. 



Remark 4.3 Note that in Definition \4-S\ the order of the columns is impor- 
tant. Note further that in the special case that A = B we have A4od(^i ^) = 
M{A) andMD{A,A) = 1. 

Then we have the following theorem. 

Theorem 4.4 Let A = [ai, . . . , a„], S = [61, . . . , 6„] G M™-'" be matrices 
with normalized columns and suppose that M£){A, B) 7^ 1. Then, 

Proof. For i ^ j, hy the triangle inequality we have that 

\{ai + bi,aj + bj)\ < \ {ai,aj) \ + \ {ai,bj) \ + \ {bi,aj) \ + \ {bi,bj)\ 

(17) 

<M{A)+ 2Mod{A, B) + M{B) 

and 

||a, + = 2 + 2{ai, h) > 2(1 - |(a„ b,)\) > 2(1 - Md{A, B)). (18) 
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Combining (fT7|) and (fTSj) . we get 

iy^i ||ai + 6i||2||ai + fej||2 ~ 2{1-Md{A,B)) 

□ 

Note that the inequahty (fT6]) also holds if = 1, if we define 

the right side to be infinite in this case. 

Remark 4.5 The bound in Theorem\4-4\ is sharp. For example if 



A 

then 



1 

1 



, B = l2 



M{A + B) = 1, M{A)=M{B)=Md{A,B) = 

and 

Mod{A,B) = 1. 

Theorem 14.41 immediately extends to more than one summand. 

Corollary 4.6 Consider matrices A^ G M'"'", \ < i < M with normalized 
columns. If 

M-2 Md{A\A^)>0, 

l<i<j<M 

then 



EZiMA') + ^El<^<J<M Mod{A\ A^] 

— 1 M-2Ei<.<,<M-^i3(^%^^) 



In order to apply these results to sums of Kronecker products of the form 
A = "^jLi . . . f?) A-^jy, we introduce the abbreviation 

. ^ E-LiMiA^) + 2Zi<.^j<MMoDiA\A^) 
M-2E,<.<,<mMd{A^,A^) 

We have the following Corollary. 

Corollary 4.7 Consider a linear system of the form 

M 



A{(^...(g) A^j^)x = b, 
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where the matrices are of appropriate dimensions and have normalized 
columns. Suppose that there exists a solution x with the sparsity 



Nlo<2 1 + 



UiEf=iA{^...^Ai,) ■ 

Then this is the unique solution with this sparsity which can be recovered 
using ii-minimization as defined in Q). 

Proof. By applying Lemma 12.71 and Corollary 14.61 we have that 



kilo < 



1 + 



Ai 



implies that 



kilo < 7^ 1 + 



M{j:r=iA{ 



A{ 



and therefore by Lemma 12.71 the sparse solution is unique. □ 

Example 4.8 Consider a sum of Kronecker products C = I ® A + A® I 
as they for example arise in the finite difference approximation of boundary 
value problems for 2D elliptic PDEs. Then it is easy to see that 

M[C) < U{C) - 2-2Mod{I®A,A0I) ' 
Especially, if A is a ID finite difference matrix, e.g. 




-1 
2 



then we have 



0.4237 = M{C)< U{C) = 0.5615. 



For the /c-restricted isometry property it is an open problem to establish 
relationships between that of a sum of Kronecker products and the sum- 
mands. 



5 Conclusion 

We have analyzed the recently introduced concepts of the spark, the mutual 
incoherence and the A;-restricted isometry property of matrix in Kronecker 
product form to that of the Kronecker factors. 
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